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ABSTRACT 

The large number of programs commercially available 
at all educational levels presents the teacher with a major selection 
problem. The purpose of this discussion is to explain the assessment 
procedures that publishers should use in order to assure the 
publication of only high quality programs and to share this 
information with educators. When he first receives a program, a 
publisher should have it checked over by an expert, and a decision 
should be rendered as to whether or not it is academically sound and 
whether it is appropriate tor a specific educational curriculum. The 
program is then objectively tested in a classroom situation. Once 
appropriate sample subjects are selected, the students are 
administered an objective achievement test as a pre-test. Next, they 
work through their programs. Attempts are made to minimize the 
effects on the students' learning of aLl variables other than the 
program itself. Upon completion ot the program, the students take the 
achievement test again as a post-test. Finally, the students and 
teachers complete standard program evaluation forms. The data 
gathered from the pre- and post-tests, from the students' actual work 
on the program, and from the evaluation forms are then analyzed. A 
final publish-no publish decision is made according to the criterion 
of the amount a student learns from the use of the program. (CK) 



£0051455 



US DEPARTMENT CF HEALTH. 
EDUCATION A WELFARE 
OFFICE Of EDUCATION 

THIS DOCUMENT HAS BEEN F1EPRO 
DUC ED EXACTLY AS RECEIVED FROM 
TH£ PERSON OR ORGANISATION OfilG 
kNATING IT POiNTS OF VIEW OR OPIN- 
IONS STATED DO NOT NECESSARUV 
REPRESENT OFFICIAL OFFICE Or EDU 
CATiGN POSITION OR POLICY 



HOWTO SELECT AND EVALUATE PROGRAMMED 
INSTRUCTIONAL M ATERI ALS 





*■ 

s ' * . ! 




i 


1 ' 


- » \ 
\ - * - 


4 





F. J. McGUIGAN 

APRIL. 197! 




ADUl'i LEARNING CENTER 




School of Educotion, North CorolL-so Stote University, Raleigh, North Carolina 

1 






ADULT LEARNING CENTER 



The Adult Learning Center was approved and established as an experimental and demonstration 
project in adult basic education in 1967 under the provisions of Section 309(b) of the Adult Education 
Act of 1966. The Center is an integral part of the research and development program of the School 
of Education at North Carolina State University. The purpose of the project is the identification, de- 
velopment, and evaluation of innovative curriculum materials and instructional methods that will ac- 
celerate and enhance learning in undereducated adults. Special emphasis has been placed upon in- 
vestigations inta the utilization of modern educational technology for the instruction of undereducated 
adults. A major concern af the Center is the development of packaged instructional materials and im- 
proved instructional methods which are capable af being institutionalized within adult basic education 
programs in public school systems. The Center is totally committed to the belief that its research 
and developmental efforts must give promise of materially improving ttv/ ongoing adult basic educa- 
tion program at all levels, local, state, and national. 

The Center is part of the program conducted under the Auspices of the Bureau of Adult, Voca- 
tional, and Technical Education, Division of Adult Education Programs, Office of Education, U. $. 
Department of Health, Education, and Welfare. The program of the Center cuts across the Schools 
of Education, Agriculture and Life Sciences, Liberal Arts, and Physical Sciences and Applied Mathe- 
matics at North Caialina State University at Raleigh. 



THE OCCASIONAL PAPER SERIES 

The Occasional Paper Series has been established by the Center as a vehicle for the publication 
of monographs af general interest ta people involved in all areas of adult education. For the most 
part, these monographs consist of papers or speeches written or delivered by members of the Center 
staff, as a consequence af their affiliation with the Center. 
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PREFACE 



In recent years there has occurred a rather dramatic prolifera- 
tion of commercially-produced programmed instructional materials. 

It is frequently the case that programmed instructional materials 
are represented as the panacea to the achievement problems of 
students of all age and ability levels. Because programmed mate- 
rials allow students to progress at their own rate of speed, 
underachievers are guaranteed eventual success in the content 
mastery of their programs. For the same reason, brighter students 
are allowed to progress as rapidly as theii abilities allow inasmuch 
as group instructional restraints have been removed from the learn- 
irg situation. 

It is indeed unfortunate that in all too many cases there is 
little or no evidence that particular programmed instructional 
materials are effective; it is not known whether or not they teach 
and, if so, under what conditions. The effectiveness of a programmed 
instructional package is directly proportional to the extent to which 
it teaches; learning outcomes provide the best measures of the effec- 
tiveness of programmed materials. 

In this report, Dr. McGuigan sets forth several guidelines which 
should be of assistance to those educators whose concern and responsi- 
bility is the identification, selection, and testing of programmed 
texts. This report is not presented as the ultimate and only state- 
ment on the subject. It is sent forth, however, with the confidence 
that it constitutes an important cor ribution to the accumulating 
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body of knowledge on programmed instruction. Its obvious utility 
is the Center's justification for its inclusion in the Occasional 
Paper Series. 

In addition to serving as a special consultant to the Alult 
Learning Center, Dr. McGuigan is Professor of Psychology at Hollins 
College, Roanoke, Virginia. 

D. Barry Lumsden, Director 
Adult Learning Center 
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HOW TO SELECT AND EVALUATE PROGRAMMED 
INSTRUCTIONAL MATERIALS 



The large number of programs commercially available at all 
educational levels presents the ceacher with a major selection 
problem — which should she choose for her students? By sharing with 
the teacher the validating data that led to his decision to publish 
a given program, the publisher can help the teacher answer her ques- 
tion. Some publishers make this type of information available in 
separate evaluation reports of the programs that it markets* The 
purpose of the present discussion is to explain the assessment proce- 
dures that publishers should use in order to assure publication of 
only high quality programs and to share this information with educators 
to the benefit of their students. We shall now consider the various 
criteria for assessing the quality of any given program. 

P re-classroom T esting Criteria 

Programming. Technique . Programmed learning has resulted in many 
products of widely varying programming quality. When a publisher 
first receives a program, it should be checked over by a rrogramming 
expert to ascertain whether or not sound principles of programming 
technique have been used by the author. Only those programs that 
have superior programming characteristics should be further considered. 

Sub ject -Mat ter- Expert Evaluation . The next step is to have the 
program thoroughly studied by one or more subject-matter experts. A 
decision is rendered as to whether or not the program is academically 
sound, whether or not the terms, principles, mechanics, etc. are 
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up-to-date, and so forth. Finally, the subject-matter expert decides 
whether or not the content of the program is appropriate for a specific 
educational curriculum. Providing that the program passes these first 
two tests, it is then objectively tested in a classroom situation. 




Assessment In A Classroom 

Efforts are made to select samples that are representative of the 
population for which the program is intended. Hence, appropriate grade 
levels ana classes are chosen, the I.Q.'s of the "experimental 11 students 
are ascertained to make sure that they are nationally typical, and so 
forth. See Appendix A for an example of a form which might be used for 
the collection of important student and student performance information. 

Once appropriate samples are selected, the students are adminis- 
tered an objective achievement test, as a pre-test. Following this 
they work through their programs, usually at their own xate. During 
this learning period attempts are made to minimize the effects on the 
students' learning of all variables other than the program itself, 
e.g., the activities of the teacher are restricted to procedural 
matters and the students work on their programs in the classroom under 
the teacher's supervision. Once the students have completed their 
programs, they take the achievement test again as a post-test. Finally, 
the students and the teachers complete standard program evaluation 
forms (See Appendices B and C) . 

The data gathered from the pre- and post-tests, from the student's 
actual work in his program, and from the evaluation forms are then 
analyzed. A final publish-no publish decision is made according to the 
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following criteria. These criteria are guidelines that the teacher 
might profitably use in helping her decide whether or not to adopt a 
given program. 

Classroom Testing Cri teria 

Learning Data , The primary purpose of the program is to teacr.-- 
if the student; does not learn a substantial amount from the use of 
the program, the other criteria may be disregarded. The question, the n, 
of how much students learn from the use of a program is much more 
important than all of the other criteria combined . 

To assess amount learned as accurately as possible we have devel- 
oped a ratio which we call G (McGuigan and Peters, 1965). To better 
understand G let us refer to Table 1. Suppose that it is possible to 
score ICO points 

Table 1 

Learning Data Analysis (Means ) 



Possible Score 
Pre-Test Score 
Post-Test Score 
Gain Score 
Possible Gain Score 

q _ Gain Score 
Possible 
Gain Score 



- 100 

- 20 

* 85 

= 85 - 20 = 65 

= 100 - 20 * 80 

- 65 = , 81 

80 



on the achievement test; in this case, the possible score is, obviously 
100. Now suppose that a class makes an average score of 20 before the 
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program was taken (the pre-test score) and of 85 upon completion of the 
program (the post-test score). The gain score can thus be seen to be 
85 minus 20 , or 65. Next we ascertain the possible gain score, i.e., 
the maximum amount of gain that the students could have demonstrated. 
This value is the possible score (100) minus the pre-test score (20), 
which is 80. Now, to compute the racio, (}, one merely needs to divide 
the possible gain score (80) into the actual gain score (65), resulting 
in this example in the value .81. 

With this understanding of how to compute the gain-to-possible-gain 
ratio, G_, let us now consider how this value helps us to answer our 
major question of whether or not the students learned a sufficient 
amount from the use of any given program. Put otnarwise: is any given 

value of G (e.g., .81) sufficiently high that we can conclude that the 
program led to an adequate amount of learning? To answer this question, 
we have constructed a frequency distribution of C scores for a number 
of programs that have been tested to date (Figure 1). Note that G 
typically varies between 0.0 and 1.0 such that the higher the G value, 
the greater the learning. By studying the G scores that make up the 
frequency distribution presented in Figure 1 we can see, for instance, 
that one program yielded a value within the .10 to .19 category, that 
two programs had G values of .20 to .29, and so forth. Now, to answer 
our question, note that the G value for our hypothetical program is 
located within this frequency distribution by means of the unshaded 
region. Since this value of .81 is a high one, relative to G resulting 
from the other programs tested, we may conclude that this program led 
to a relatively superior amount of learning. 
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Student Evaluations . It has been shown that student reactions 



are positively related to amount learned (McGuigan ar d Peters, 1965) 
and, in any event, one should prefer using programs that students 
regard as enjoyable and beneficial. The standard student evaluation 
form includes six items on which students rate the program. In using 
this criterion, we compare the student reactions to a test program 
with all student reactions to previous programs that have been tested. 
That is, we sum the student reactions for all programs tested to date 
and then compare their reactions to the program currently undergoing 
testing (’fcGuigan and Peters, 1965). Consider Table 2 as an example. 
We can see that 68% of the 1489 students who studied a variety of pro- 
grams indicated that ttiey learned more because a program was used in 
their course. In contrast, 81% said that they learned more because 
the test program was used. We may thus regard the student Reactions 
to the test program in this example as being relatively favorable. 

Each report of the validating data for any given program presents 
these kinds of findings in detail. 



Table 2 

Student Reactions To Test Program 

Question //l: Because a program was used in this course, I believe: 





N 


I learned more 


It made no difference 


I learned less 


31 Programs 


1489 


68% 


19% 


13% 


Test Program 


59 


81% 


7% 


12% 




Teacher Evaluation . The teachers' reactions, as recorded on 
standard evaluation forms, are verbally summarized in each report. 
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Error Analysis . Research has also indicated that the higher the 
error rate, the less the learning ^McGuigan and Peters, 1965). This 
finding is consistent with principles of learning (especially of the 
Skinnerian variety). Hence, both empirical and theoretical considera- 
tions indicate that the error rate for a given program should be 
relatively low. To ascertain the mean error rate, the responses that 
the students wrote in their pregrams are carefully checked and the 
number of errors is counted for each student. The mean number of 
errors is then determined for the entire sample of student 3, and this 
value is divided b> the total number of responses called for by the 
program. The resulting value is the mean error rate. Figure 2 
presents a frequency distribution of mean error rates for programs 
that have been previously tested. Figure 2 can now be used to deter- 
mine whether or not the error rate for a given program is excessive. 
For example, suppose that a program yielded a mean error rate of 2.3%. 
This value, represented by the unshaded region in Figure 2, can be 
seen to be a low one, relative to the mean error rates for the entire 
sample of programs. 



In summary, then, one who is considering adopting any given 
program sho.ild first request and examine the learning data made 
available by the publisher. Program and learning data obtained 
from publishers should, as much as possible, conform to the 
Recomrenda t i ons for Repo rtlng the hffectivene ss of Progra med 
Instruction Material prepared by the Joint Committee on Programmed 
instruction and Teaching Machines. If the information obtained 



Concluding Statement 
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indicates that the students learned a considerable amount from the 



use of the program, consideration may then be given to the more 
subsidiary criteria, i.e., has the content of the program received 
the approval of subject-matter experts, are the teacher and student 
evaluations of the program favorable, and does it have a low error 
rate? The program that best satisfies these criteria should be the 
one favored for adoption. 
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APPENDIX A 



STUDENT TIME LOG AND GENERAL INFORMATION FORM 



Title of Program 


Sex 




Name of Student 


Race 




Highest Grade Completed 


Age 




Tested Reading Level 


I. Q. 





Date & Day 



Study Time In Hours And Minutes : 

Time Started Time Stopped 
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APPENDIX B 



1EACHER EVALUATION FORM 

Title of Program Name of Teacher 

The following questions were designed to help us evaluate the 
program that you have just gone through with your class. The infor- 
mation that you can furnish will be of great value to us. For each 
question please check the blank that you feel most adequately describes 
your opinion. Blank lines have been provided below each question for 
you to qualify or elaborate your answers. Please feel free to make any 
comments that will aid us in determining the value of this program. 

1. Is the subject-matter of the program academically sound? 

Yes 

No 

_ Undecided 

Comments : 



2. Was the level of the subject matte^ appropriate for your class? 

Too difficult 

Appropriate 

Too easy 
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Comments : 



3. As contrasted with what you have been able to accomplish with other 
types of learning material, how much do you feel you were able to 
get your pupils to learn with this program? 

A great deal more than with most other materials. 

A little more than with most other materials. 

About as much as with other materials. 

A little less than with most other materials. 

So little as to be a waste of time. 

Comments ; 



4. The next time you teach a course in this subject or a similar 
field, would you: 

Prefer to have programs used for at least part of 

the course? 

Prefer not to have programs used? 

Not care whether programs are used or not? 

Comments I 



5. To what extent did you enjoy using this program with your class? 



Very Unenjoyable 50-50 Enjoyable Very 

Unenjoyable Enjoyable 

Comments : 
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6. Do you think this program should be made available for the use of 



teachers throughout the country? 

Ves 

No 

Do^t know 

Comments : 



7. 



In your own words, would you please summarize your opinion of 
this program. Include statements about its strong and weak points. 
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APPENDIX C 



STUDENT EVALUATION FORM FOR 



1. Because a program was used in this course, I believe: 

I learned more. 

it made no difference. 

I learned less. 

2. In comparing work done using the program with studying in regular 
textbooks, I feel that, with the sane amount of time and effort: 
I learned much more with the program. 

I learned somewhat more with the program* 

there is no difference. 

I learned somewhat more from studying textbooks. 

I learned much more from studying textbooks. 

3. If I were to take another course in this subject or a similar 
field, I would : 

prefer to have programs used for at least part of the course. 

prefer not to have programs used. 

not care whether programs are used or not. 

4. How much do you think you learned from this program? 



Learned Learned Learned Learned Learned 

nothing a little a medium quite a bit very much 

amount 

5. To what extent did you enjoy going through this program? 



Unenjoyable 



50-50 Enjoyable 
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Very 

Unenj oyable 



Very 

Enjoyable 



6. To what extent was the program repetitious? 



Much too Too Moderately Slightly Not at all 

repetitious repetitious repetitious repetitious repetitious 

7. In your own words say what you thought of the program. For example 

what did you like about the program? What did you dislike about it 



etc . ? 
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